{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# imagenette_experiments\n",
    "> experiments with fastai imagenette / imagewoof datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks with experiments. Models trained with fastai v1 on Imagewoof dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This results was done with experimental model - XResnet with modification.  \n",
    "I used pool layer plus convolution stride 1 instead of convolution stride 2.  \n",
    "And instead of regular pytorch pool (AveragePool2d and MaxPool2d) i used MaxBlurPool as described here:   \n",
    "[fastai forum topic](https://forums.fast.ai/t/imagenette-imagewoof-leaderboards/45822/20?u=a_yasyrev)  \n",
    "[github ducha-aiki](https://github.com/ducha-aiki/Ranger-Mish-ImageWoof-5/blob/master/mxresnet.py#L121)  \n",
    "\n",
    "Activation function - Mish, long disscussion [on fastai forum](https://forums.fast.ai/t/meet-mish-new-activation-function-possible-successor-to-relu)  \n",
    "Fit with Ranger optimizer and flat with annealing - [long tread on fastai forum](https://forums.fast.ai/t/how-we-beat-the-5-epoch-imagewoof-leaderboard-score-some-new-techniques-to-consider)  \n",
    "\n",
    "Model was created with [model-constructor](https://github.com/ayasyrev/model_constructor)  \n",
    "Explanation how model was created here: https://github.com/ayasyrev/imagenette_experiments/blob/master/ResnetTrick_create_model_fit.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Current results vs results on leaderboard - ImageWOOF:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Size (px) | Epochs |   | Accuracy | # Runs | My res | URL |  Comments |\n",
    "|--|--|--|--|--|--|--| -- |\n",
    "|128|5|  |73.37%|5, mean| |\n",
    "|128|20||85.52%|5, mean|86.10% | |\n",
    "|128|80||87.20%|1| 87.63% |[notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s128_e80_8763.ipynb) | 3 runs, start_pct=0.3\n",
    "|128|200||87.20%|1|  88.30%| [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s128_e200_8830.ipynb) | 3 runs, start_pct=0.2\n",
    "||||||| \n",
    "|192|5||75.94%|5, mean| 77.87% | [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s192bs32.ipynb) | added to board\n",
    "|192|20||87.25%|5, mean| 87.85% | [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s192bs32.ipynb)  | added to board\n",
    "|192|80||89.21%|1| 89.69% |[notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s192bs32_e80_8969.ipynb) | 4 runs.\n",
    "|192|200||89.54%|1| 90.35% |[notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s192bs32_e200_9035.ipynb) | 2 runs.\n",
    "|||||||\n",
    "|256|5||76.87%|5, mean| 78,84% | [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s256bs32.ipynb)| added to board\n",
    "|256|20||88.29%|5, mean| 88,58% | [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s256bs32.ipynb)| added to board\n",
    "|256|80||90.48%|1| 90.63% | [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s256bs16_e80_9063.ipynb)| 2 runs, start_pct=0.4\n",
    "|256|200||90.38%|1| 91.14% | [notebook](https://github.com/ayasyrev/imagenette_experiments/blob/master/Woof_MaxBlurPool_ResnetTrick_s256bs16_e200_9114.ipynb)| 3 runs, start_pct=0.2\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to repeat results:  \n",
    "All notebooks can be run on google colab.  \n",
    "\n",
    "Install this repo:\n",
    "` pip install -e git+https://github.com/ayasyrev/imagenette_experiments`  \n",
    "Now import Model constructor and helper utils as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imagenette_experiments.train_utils import *\n",
    "from imagenette_experiments.trick_model import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create model constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check model, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NewResBlock(\n",
       "  (reduce): MaxBlurPool2d()\n",
       "  (convs): Sequential(\n",
       "    (conv_0): ConvLayer(\n",
       "      (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (act_fn): Mish()\n",
       "    )\n",
       "    (conv_1): ConvLayer(\n",
       "      (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (act_fn): Mish()\n",
       "    )\n",
       "    (conv_2): ConvLayer(\n",
       "      (conv): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (idconv): ConvLayer(\n",
       "    (conv): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "    (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  )\n",
       "  (merge): Mish()\n",
       ")"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.body.l_1.bl_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets create Learner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data path   /root/.fastai/data/imagewoof2\n",
      "Learn path /root/.fastai/data/imagewoof2\n"
     ]
    }
   ],
   "source": [
    "learn = get_learn(woof=1, size=128, bs=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we cat train it regular fastai way, for example fit with annealing:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`learn.fit_fc(tot_epochs=5, lr=1e-4, moms=(0.95,0.95), start_pct=0.72)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
